Export fp8 te nemo to trt-llm #10096
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
added 3 commits
August 14, 2024 14:34
Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com>
Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com>
Laplasjan107
force-pushed
the
export_fp8_te_nemo_to_trtllm
branch
from
eaef189
to
4816a6e
Compare
added 2 commits
August 14, 2024 14:46
Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com>
Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com>
Laplasjan107
force-pushed
the
export_fp8_te_nemo_to_trtllm
branch
from
4816a6e
to
042d325
Compare
Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com>
Signed-off-by: Laplasjan107 <Laplasjan107@users.noreply.github.com>
Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com>
janekl
reviewed
Aug 19, 2024
janekl
reviewed
Aug 19, 2024
Signed-off-by: Piotr Kamiński <67481570+Laplasjan107@users.noreply.github.com>
Signed-off-by: Laplasjan107 <Laplasjan107@users.noreply.github.com>
Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com>
…lasjan107/NeMo into export_fp8_te_nemo_to_trtllm
Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com>
Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com>
Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com>
oyilmaz-nvidia
requested changes
Aug 22, 2024
There was a problem hiding this comment.
Can we please remove the parts that are not related to the fp8 ckpt support? I saw some of the values in the config removed. We are in the process of moving main parts of the export into mcore and some of the code optimizations were done in the mcore already.
JimmyZhang12
previously approved these changes
Aug 22, 2024
There was a problem hiding this comment.
In general most of the changes are very much well needed code cleanups that had needed to happen for a while.
@shanmugamr1992 is simultaneously working to port this code to megatron core, and he also has many changes to cleanup the code. As @oyilmaz-nvidia has mentioned, we should coordinate with him to see which of these are needed to land in this PR For instance he also refactors the giant messy weight name dict here.
Can we make sure these refactors do not break the NeMo-Aligner's existing code path? I believe the CI isn't running for aligner. cc @terrykong
|
@JimmyZhang12 I'm not sure I will be able to test this, but I'd like to try to manually test the NeMo-Aligner TRTLLM integration with this PR. It looks like this PR is rooted after @oyilmaz-nvidia upgraded to v11, so I can try a one-off build of (TRTLLM v10) + (this Nemo PR) + (mcore ToT) and hopefully with a few hacks I can validate |
Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com>
Signed-off-by: Laplasjan107 <Laplasjan107@users.noreply.github.com>
terrykong
requested changes
Aug 26, 2024
terrykong
requested changes
Aug 26, 2024
Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com>
Signed-off-by: Laplasjan107 <Laplasjan107@users.noreply.github.com>
Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com>
terrykong
requested changes
Aug 27, 2024
…lasjan107/NeMo into export_fp8_te_nemo_to_trtllm
Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com>
Signed-off-by: Laplasjan107 <Laplasjan107@users.noreply.github.com>
terrykong
approved these changes
Aug 28, 2024
oyilmaz-nvidia
approved these changes
Aug 28, 2024
janekl
approved these changes
Aug 28, 2024
Edresson
pushed a commit
that referenced
this pull request
Aug 29, 2024
* initial commit Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * PR draft Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * fixed scaling weights Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * Apply isort and black reformatting Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * Apply isort and black reformatting Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * fixed zarr loading, added flags, refactor Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * Apply isort and black reformatting Signed-off-by: Laplasjan107 <Laplasjan107@users.noreply.github.com> * fix expert key mapping Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * refactor Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * Apply isort and black reformatting Signed-off-by: Laplasjan107 <Laplasjan107@users.noreply.github.com> * fix: failed test was finishing with exit code 0 Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * test commit -- rerun github checks Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * bugfix: naming Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * bugfix v2: naming Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * apply code review changes Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * Apply isort and black reformatting Signed-off-by: Laplasjan107 <Laplasjan107@users.noreply.github.com> * fix TensorRTLLM build (fp8 still not supported) Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * Apply isort and black reformatting Signed-off-by: Laplasjan107 <Laplasjan107@users.noreply.github.com> * undo refactor Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * bugfix: arguments to dist_convert Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * Apply isort and black reformatting Signed-off-by: Laplasjan107 <Laplasjan107@users.noreply.github.com> --------- Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> Signed-off-by: Laplasjan107 <Laplasjan107@users.noreply.github.com> Signed-off-by: Piotr Kamiński <67481570+Laplasjan107@users.noreply.github.com> Co-authored-by: Piotr Kaminski <pikaminski@nvidia.com> Co-authored-by: Laplasjan107 <Laplasjan107@users.noreply.github.com> Signed-off-by: Edresson Casanova <edresson1@gmail.com>
adityavavre
pushed a commit
to adityavavre/NeMo
that referenced
this pull request
Sep 15, 2024
* initial commit Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * PR draft Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * fixed scaling weights Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * Apply isort and black reformatting Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * Apply isort and black reformatting Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * fixed zarr loading, added flags, refactor Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * Apply isort and black reformatting Signed-off-by: Laplasjan107 <Laplasjan107@users.noreply.github.com> * fix expert key mapping Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * refactor Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * Apply isort and black reformatting Signed-off-by: Laplasjan107 <Laplasjan107@users.noreply.github.com> * fix: failed test was finishing with exit code 0 Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * test commit -- rerun github checks Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * bugfix: naming Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * bugfix v2: naming Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * apply code review changes Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * Apply isort and black reformatting Signed-off-by: Laplasjan107 <Laplasjan107@users.noreply.github.com> * fix TensorRTLLM build (fp8 still not supported) Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * Apply isort and black reformatting Signed-off-by: Laplasjan107 <Laplasjan107@users.noreply.github.com> * undo refactor Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * bugfix: arguments to dist_convert Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> * Apply isort and black reformatting Signed-off-by: Laplasjan107 <Laplasjan107@users.noreply.github.com> --------- Signed-off-by: Piotr Kaminski <pikaminski@nvidia.com> Signed-off-by: Laplasjan107 <Laplasjan107@users.noreply.github.com> Signed-off-by: Piotr Kamiński <67481570+Laplasjan107@users.noreply.github.com> Co-authored-by: Piotr Kaminski <pikaminski@nvidia.com> Co-authored-by: Laplasjan107 <Laplasjan107@users.noreply.github.com> Signed-off-by: adityavavre <aditya.vavre@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Add support for exporting FP8 TE NeMo to TRT LLM.
Collection: nlp
Changelog
Usage
GitHub Actions CI
The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.
The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".
Before your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information